We are seeking an experienced Azure Databricks Architect to design and implement scalable big data solutions leveraging the Microsoft Azure cloud ecosystem. The ideal candidate will possess deep technical knowledge of Databricks, data engineering best practices, and modern cloud architectures. This role will be instrumental in leading data transformation initiatives, optimizing data pipelines, and ensuring secure, high-performance environments for advanced analytics and AI workloads.
Architect, design, and deploy Azure Databricks solutions tailored to business requirements and aligned with best practices.
Develop end-to-end data pipelines using Spark (PySpark/Scala) in Databricks.
Integrate Databricks with Azure services including Data Lake Storage Gen2, Synapse, Key Vault, and Data Factory.
Optimize and tune Spark jobs for performance and cost-efficiency.
Design secure, scalable environments with appropriate networking, RBAC, and identity management.
Lead architecture discussions and decisions with cross-functional teams, including data engineers, analysts, and DevOps.
Define and implement CI/CD pipelines for Databricks notebooks and infrastructure using tools such as Azure DevOps or GitHub Actions.
Create documentation and reusable assets for deployment standards and architecture patterns.
Stay up to date with new Azure features and recommend adoption strategies.
Required:
7+ years of experience in data architecture or engineering.
3+ years hands-on experience with Azure Databricks and Apache Spark.
Strong experience with Azure cloud services: ADF, ADLS, Synapse, Key Vault, Event Hubs, etc.
Proficiency in PySpark, SQL, and/or Scala.
Deep understanding of big data design patterns, ETL/ELT strategies, and data governance.
Strong knowledge of CI/CD practices for data workloads.
Familiarity with security best practices for cloud-based environments.
Preferred:
Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect).
Experience with Delta Lake, MLflow, and Databricks Unity Catalog.
Exposure to machine learning pipelines or data science workflows.
Background in enterprise data warehouse modernization projects.
All qualified applicants will receive consideration for employment without regard to race, color, national origin, age, ancestry, religion, sex, sexual orientation, gender identity, gender expression, marital status, disability, medical condition, genetic information, pregnancy, or military or veteran status. We consider all qualified applicants, including those with criminal histories, in a manner consistent with state and local laws, including the California Fair Chance Act, City of Los Angeles' Fair Chance Initiative for Hiring Ordinance, and Los Angeles County Fair Chance Ordinance. For unincorporated Los Angeles county, to the extent our customers require a background check for certain positions, the Company faces a significant risk to its business operations and business reputation unless a review of criminal history is conducted for those specific job positions.